feat(btrblocks): pick per-chunk bit widths when fastlanes.bitpacked_v2 is allowed - #9754
feat(btrblocks): pick per-chunk bit widths when fastlanes.bitpacked_v2 is allowed#9754mhk197 wants to merge 2 commits into
Conversation
5239669 to
f3fd087
Compare
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | arrow_checked_add_u32_avx512[16384] |
17.6 µs | 21.3 µs | -17.1% |
| ⚡ | WallTime | arrow_checked_add_u32_neon[16384] |
20.4 µs | 12.7 µs | +59.84% |
| ⚡ | WallTime | mul_i32_nullable_avx512 |
9.6 µs | 8.7 µs | +10.49% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing mk/bitpacked-v2-editions (114a0f0) with mk/bitpacked-v2 (dcd7e85)
Footnotes
-
206 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
b978a04 to
9d950f7
Compare
9d950f7 to
01b289f
Compare
01b289f to
6258f9d
Compare
|
|
||
| #### `preview2026.09.0` | ||
|
|
||
| - `array`: `fastlanes.bitpacked_v2` |
There was a problem hiding this comment.
just a caution that preview means that we are 99% certain that this is the right serialisation format
| /// common case of an encoding with a single wire format. A scheme whose output requires a | ||
| /// newer format of an encoding lists that format here, so a writer restricted to editions | ||
| /// without it drops the scheme instead of producing arrays it cannot serialize. | ||
| fn produced_serialized_ids(&self) -> Vec<ArrayId> { |
There was a problem hiding this comment.
@joseph-isaacs and I convinced ourselves that this is unnecessary. The logic is that in old edition there's only old compressor, in new edition there's the new compressor. Then upon serialisation on old edition you will produce bitpacked v1 because that's what the compressor produced (i.e. you can unwrap v2 to v1) and in the new edition you will produce v2. If unwrapping is not possible it's a different encoding.
I think we need a better way to configure the compressor...
There was a problem hiding this comment.
here its not always possible to downgrade v2 -> v1 though right? if there are different bit widths per chunk
There was a problem hiding this comment.
actually nvm, if new edition is disabled an array written with v1 has to round trip (disk) v1 -> (mem) v2 -> (disk) v1 unless there's been some intermediate transformation
|
I think you want single compressor scheme if you have single in memory array. Then if v2 is allowed we produce v2, otherwise produce v1 |
6258f9d to
9f9fc0c
Compare
CascadingCompressor carries the serialized IDs its output may use. The file writer passes the same set it gives the array context, next to the existing filter on allowed encodings, so a scheme whose encoding has several wire formats can produce the newest one still allowed without the compressor knowing about editions. The CUDA preset allows only each encoding's original format, since it has no per-chunk bit-unpacking kernel yet. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…2 is allowed BitPackingScheme chooses a width per 1024-element chunk when the compressor may use the v2 format and one global width otherwise, so an old edition keeps writing the original format and a newer one gets per-chunk widths from the same scheme. The width table child is re-encoded through the cascade. The scheme returns the original array when half or more of the values would be patches. A few wide values otherwise pack at width 0 with nearly every value patched, which beats raw storage by a couple of buffer bytes and loses them back in footer metadata. No shipped edition permits the format yet; the writer test declares its own. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
9f9fc0c to
114a0f0
Compare
Summary
Stacked on #9750. Lets the writer emit
fastlanes.bitpacked_v2when its editions permit it. A compressor scheme looks at what is enabled and produces the newest available format.CascadingCompressor::with_allowed_serialized_ids/allows_serialized_id. The compressor carries the serialized IDs its output may use. The file writer passes the same set it gives the array context, right where it already filters schemes by allowed encodings, and the compressor knows nothing about editions themselves. Without a restriction, which is whatBtrBlocksCompressor::default()gives in-memory callers, every ID is allowed and the newest format wins.BitPackingSchemeproduces the newest allowed format. It picks a width per 1024-element chunk whenfastlanes.bitpacked_v2is allowed and one global width otherwise, so an old edition keeps writing the original format and a newer one gets per-chunk widths from the same scheme. There is no second scheme and no per-scheme wire-ID declaration. The width table child is re-encoded throughcompress_child, like the children of the decimal and temporal schemes.Tests
core_writer_never_emits_bitpacked_v2: a session with the default encodings and only the core edition enabled writes a column whose chunks need 1 to 22 bits, and every array it produces reads back asfastlanes.bitpacked.permitting_writer_emits_bitpacked_v2: the same session plus a test-local edition permitting the format writes the column asfastlanes.bitpacked_v2and it reads back equal.cuda_compatible_disallows_per_chunk_bitpacking: the default compressor allows the format and the CUDA preset does not.test_mostly_patched_stays_primitive: six i64 values, five of them wide, stay primitive instead of becoming width 0 with five patches.new_array_contexttests cover the allowed set: the editions' serialized IDs, or every registered ID when editions are disabled.vortex-btrblocksgolden snapshots show per-chunk widths and awidth_tablechild for bit-packed columns in both the default andunstableconfigurations; the four default goldens that change all shrink.Validation
cargo nextest run -p vortex-compressor -p vortex-file -p vortex-edition -p vortex-btrblocks -p vortex editions, with and without--features unstable_encodingscargo clippy --all-targets --all-featureson vortex-compressor, vortex-file, vortex-edition, vortex-btrblocks, andvortexwithunstable_encodings;cargo +nightly fmt --all